Papers with RL fine-tuning
Unveiling the Implicit Toxicity in Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies focus on probing toxic outputs that can be easily detected with existing toxicity classifiers, but LLMs can generate diverse implicit toxic output that are difficult to detect via simply zero-shot prompting. |
| Approach: | They propose a reinforcement learning based attacking method to induce the implicit toxic outputs in large language models by fine-tuning toxicity classifiers. |
| Outcome: | The proposed method generates implicit toxic outputs that are difficult to detect via zero-shot prompting on five widely-adopted toxicity classifiers. |
Controlling Multimodal Conversational Agents with Coverage-Enhanced Latent Actions (2026.acl-long)
Copied to clipboard
| Challenge: | Recent reinforcement learning (RL) has been widely explored for adapting MCAs to various human-AI interaction scenarios. |
| Approach: | They propose to use a latent action space for reinforcement learning instead of RL to fine-tune MCAs. |
| Outcome: | The proposed method outperforms baselines on two conversation tasks with a novel cycle consistency loss. |
Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent approaches to language model alignment assume homogeneous human preferences, but actual human preferences vary widely and are hard to satisfy with a single language model. |
| Approach: | They propose an RL-free extension of Direct Preference Optimization (DPO) that folds language modeling directly into reward modeling and trains language models as collective reward models that combine all objectives with specific weights. |
| Outcome: | The proposed method matches or outperforms existing methods in safety alignment and long-form question answering. |
ToM-Synth: Scaling Robust Theory of Mind in LLMs via 6,912 Structured Social Units (2026.findings-acl)
Copied to clipboard
Guiyang Hou, Xiang Huang, Shangke Lyu, Yuchuan Wu, Weiyao Luo, Xinyu Mei, Yongliang Shen, Weiming Lu, Yongbin Li
| Challenge: | Existing methods endowing LLMs with Theory of Mind fail to internalize the augmented ToM into the LLM. |
| Approach: | They propose a factorial combinatorial synthesis framework that enables systematic synthesis of ToM data and uses it for RL fine-tuning. |
| Outcome: | The proposed framework yields a training dataset of 27,648 instances. |